Systems and methods for detecting obstructions in a camera field of view

ABSTRACT

A system mounted on a vehicle for detecting an obstruction on a surface of a window of the vehicle, a primary camera is mounted inside the vehicle behind the window. The primary camera is configured to acquire images of the environment through the window. A secondary camera is focused on an external surface of the window, and operates to image the obstruction. A portion of the window, i.e. window region is subtended respectively by the field of view of the primary camera and the field of view of the secondary camera. A processor processes respective sequences of image data from both the primary camera and the secondary camera.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC 119(e) from U.S.provisional application 60/738,991 filed Nov. 23, 2005, the disclosureof which is included herein by reference.

FIELD OF THE INVENTION

The present invention relates to detecting and classifying obstructionson the windshield of an automobile as part of a vehicle control systemand more particularly the present invention detects and identifiesobstructions on the windshield using a camera focused on the plane ofthe windshield.

BACKGROUND OF THE INVENTION AND PRIOR ART

Cameras are often used to acquire one or more images while viewingthrough a transparent window such as of glass. One such application isillustrated in FIG. 1, a vehicle control system 80 as disclosed by Steinin U.S. Pat. No. 7,113,867. A camera 32 is mounted inside a “host”vehicle 10 behind the windshield, and camera 32 views the environmentthrough the windshield. System 80 includes camera 32 and a processor 60that analyzes the images acquired by camera 32. System 80 is operativeto detect lane markings in road 20, pedestrians, other vehicles orobstacles, road signs. Commonly, the windshield becomes less transparentdue to rain, condensed moisture and/or dirt and image quality of theimages produced by camera 32 may be impaired.

U.S. Pat. No. 6,392,218, by Kuehnle, discloses a method for analyzingoptical images to detect moisture on a surface such as a windshield ofan automobile using a sensor that is mounted on the windshield of thevehicle. An image such as that of the hood of the vehicle is processedand blurriness in the image is associated with moisture on thewindshield. The sensor includes an optical image detector and amicroprocessor with supporting control circuitry. The optical imagedetector acquires two-dimensional optical images from either the vehicleitself or an area surrounding the vehicle (e.g. the hood, the road aheadof the vehicle). The acquired optical images are then delivered to themicroprocessor, which analyzes the image to determine whether moisture(such as rain) exists on the windshield. Upon detection of sufficientmoisture on the vehicle, the microprocessor circuitry can activate thewindshield wipers. U.S. Pat. No. 6,392,218 is directed towards onlydetection of rain or moisture on the windshield.

U.S. Pat. No. 5,923,027, by Stam et al, discloses a method of detectingmoisture on a surface of a windshield using a sensor in conjunction withan infrared LED. Edges of moisture (i.e. rain) are detected by thesensor. The LED is used to check for fog on the windshield based uponthe manner in which light is reflected. All objects detected by thesensor, including bugs, dirt, and headlamps from oncoming vehicles whichcreate a bright spot in the image would initially be interpreted asmoisture which then causes the windshield wipers to be activated. It isonly after the wipers have been activated, if the object remains on thewindshield, is the obstruction flagged as something other than rain. Thespontaneous, random motion of the wipers creates an unnecessary drivingdistraction for the driver. Additionally, this system cannotdifferentiate between fog on the inside of the window and fog on theoutside of the window.

U.S. Pat. No. 6,768,422, by Scofield, et al, discloses a method fordetermining if there is moisture on the surface of a windshield using asensor in conjunction with a polarizing filter. The sensor determines ifrain is present by using an edge detection algorithm. Fog is checked forby the polarization patterns created in the image.

All of the above previously disclosed patents are limited by not beingable to differentiate between various objects which might appear on awindshield of a vehicle—including, rain, dirt, dust, frost, bugs,cracks, and bird droppings. Additionally these systems are not able todistinguish between fog on the inside of the window and fog on theoutside of the window.

None of the above prior art systems provide a solution to bright spotsof oncoming vehicles, being misinterpreted as moisture.

Just as a driver of a vehicle responds differently to variousobstruction on a windshield, e.g., turning on wipers, spraying thewindows, enabling the defogger, etc, so to there is a need for and itwould be advantageous to have a vehicular vision system capable ofdetecting and distinguishing between different obstructions, e.g.,between moisture and dirt on the windshield of a vehicle, as well asbeing able to distinguish whether the obstruction is on the inside ofthe window, e.g., fog, smoke or dust, or on the outside of the vehicle,e.g., fog, frost, snow. And in order to minimize extraneous use of thewipers there is a need to for a vehicular vision system which is capableof differentiating between bright sots from on-coming vehicles andmoisture on the window.

DEFINITIONS

The term “primary camera” as used herein refers to camera 32 mountedinside a vehicle and used in systems for detecting road lane markings,other vehicles, pedestrians, and/or road signs. Typically, primarycamera 32 focuses on objects that are over five meters from the primarycamera.

The term “secondary camera” as used herein refers to a camera mountedinside a vehicle and used for detecting obstructions on the windshield,according to the present invention. Typically, the secondary camerafocuses on objects that are less than 20 cm from the camera.

The term “edge” as used herein refers to a series of adjacent points ina digital image at which the intensity as measured at the points changesnoticeably. An edge may be caused by different phenomena including: (a)discontinuities in depth, (b) discontinuities in surface orientation and(c) variations in scene illumination.

The term “gradient” as used herein refers to a two dimensional vectorwith the components given by the derivatives in the horizontal andvertical directions, of the image intensity function at each imagepoint. At each image point, the gradient vector points in the directionof largest possible intensity increase, and the length or magnitude ofthe gradient vector corresponds to the rate of change in that direction.

The term “windshield” as used herein refers to a window either flat orcurved installed on a vehicle which is at least partially transparent ina portion of the visible spectrum but not significantly translucent,opaque or hazy.

The terms “window” and “windshield” are used herein interchangeably andrefers to the front window of the vehicle, the back window or a sidewindow.

The term “windshield region” as used herein refers to the area of awindshield of a vehicle, for example the windshield, subtended by thefield of view of a camera situated inside the vehicle.

The terms “classify” and “identify” are used herein interchangeably inthe context of classifying or identifying obstructions on a window.

SUMMARY OF THE INVENTION

According to the teachings of the present invention there is provided asystem mounted on a vehicle for detecting an obstruction on an externalsurface of the windshield of the vehicle. A primary camera is mountedinside the vehicle; the primary camera is configured to acquire imagesof the environment through the windshield. A secondary camera is focusedon an external surface of the windshield, and operates to image theobstruction. A portion of the windshield, i.e. windshield region issubtended respectively by the field of view of the primary camera andthe field of view of the secondary camera. A processor processesrespective sequences of image data from both the primary camera and thesecondary camera.

In vision systems where the image analysis is at least partially basedon edge detection, such edges are represented in the images by highspatial frequencies. If the windshield is sufficiently clean, no edgesor only edges with insignificant gradient will show in the images of thesecondary camera.

According to the present invention there is provided a computerizedsystem including a camera mounted on a vehicle for classifying at leastone obstruction on a surface of a window of the vehicle, the systemincluding: (a) a secondary camera mounted inside the vehicle, thesecondary camera focusing on at least a portion of the surface of thewindow, and (b) a processor which processes a plurality of secondaryimages acquired by the secondary camera, and identifies the obstructionwhen present on the surface. The surface of the window can be selectablyeither the external surface of the window or the internal surface of thewindow.

The system further includes a primary camera mounted inside the vehicle,the primary camera configured to acquire multiple primary images of theenvironment through the window. A region of the window is subtended byrespective fields of view of the secondary camera and the primarycamera. The processor processes respectively at least one of the primaryimages and at least one of the secondary images.

The processor can use information extracted from the at least oneprimary image to selectably either enhance identifying the at least oneobstruction or eliminating false identification of an obstruction.

The obstruction may cause a deterioration of the quality in the primaryimage. The processor is capable of determining the deterioration ofimage quality in the primary images.

The system may further include an activation mechanism operativelyattached to the processor. The activation mechanism causes cleaningfluid to be sprayed on the window and windshield wipers to be activatedwhen the processor identifies an obstruction, including dirt on thewindow while the surface is dry. When the obstruction includes dirt andmoisture the processor activates the wiper.

When the processor identifies an obstruction, the processor initiates atask to remove the obstruction. The task is selected from the group oftasks consisting of: notification of the driver of the vehicle,disabling the system, activating wipers with or without cleaning fluidand activating defogging, turning on fog lights.

The secondary camera may also focus on at least a portion of theinternal surface of the window and identify an obstruction when presenton internal surface.

The secondary camera may perform tasks for the primary camera selectedfrom the group of applications consisting of: ambient light sensing,gain control sensing.

The system may further include a pulsing light source, capable ofilluminating a region of the window. The secondary camera issynchronized with the pulsing of the light source and images theilluminated region.

According to the present invention there is provided a method forclassifying at least one obstruction on a surface of a window of avehicle, using a computerized system including a camera mounted on thevehicle for, the method including: (a) providing a secondary cameramounted inside the vehicle focusing on a surface of the window, andproviding a secondary image of the surface; and (b) processing thesecondary image thereby identifying an obstruction when present on thesurface. The method may include edge detection to detect theobstruction.

The method may further include providing a primary camera mounted insidethe vehicle, configured to acquire multiple primary images of theenvironment through the window. A region of the window is subtended byrespective fields of view of the secondary and the primary camera. Theprocessing of a primary image yields detecting at least one obstructionand determining deterioration of image quality in the primary image,which is caused by the obstruction.

Identifying an obstruction is also performed by processing a primaryimage which may selectably either enhance the identification of theobstruction or eliminating a false identification of an obstruction.

The method may use the secondary camera ambient light sensing set themode of the system to selectably either day mode, dusk mode or nightmode and optionally, when determining the level of ambient light asbeing of dusk mode or night mode, turning on the headlights.

The method can identify the obstruction is being one in the groupconsisting of: rain, dirt, mud, snow, frost, cracks, smoke, dust,condensation, smeared insects and bird droppings. Upon identifying anobstruction, the method initiates a task to remove the obstruction. Thetask is selected from the group of tasks consisting of: notification ofthe driver of the vehicle, disabling the system, activating windshieldwipers, spraying fluid on windshield and activating defogging of awindow.

The processing of a primary image may include at least one othervehicular vision processing application selected from the group ofapplications consisting of: detecting lane markings in a road, detectingpedestrians, detecting vehicles, detecting obstacles, detecting roadsigns, lane keeping, headway keeping and headlights control. Theapplication may be turned off, when an obstruction is identified.

The identifying of the obstruction may include classifying theobstruction, using a Support Vector Machine (SVM) techniques or SVM withRadial Basis Function kernel techniques or any other technique know inthe art.

The method may further include illuminating the window with a pulsinglight source, whereas the secondary camera is operated synchronouslywith the pulsing light source.

These and other advantages of the present invention will become apparentupon reading the following detailed descriptions and studying thevarious figures of the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will become fully understood from the detaileddescription given herein below and the accompanying drawings, which aregiven by way of illustration and example only and thus not limitative ofthe present invention, and wherein:

FIG. 1 illustrates a prior art system installed in a vehicle having aprimary camera, designated to detect road lane markings, pedestrians,other vehicles or obstacles, and/or road signs;

FIG. 2 illustrates a vehicle having a camera obstruction detection andclassification system, in accordance with an embodiment of the presentinvention;

FIG. 3 is a perspective view that exemplifies a camera obstructiondetecting and classifying system where the primary camera and thesecondary camera are set side by side;

FIG. 4 is a perspective view that exemplifies a camera obstructiondetection system where the primary camera and the secondary camera areset one below the other;

FIG. 5 is a close up side view illustrating a camera obstructiondetection system where the primary camera and the secondary camera areset one below the other, behind the vehicle obstructed windshield;

FIG. 6 is a schematic flow diagram showing the data and decision flow ina camera obstruction detecting system, in accordance with an embodimentof the present invention;

FIG. 7 is a schematic flow diagram outlining an algorithm thatexemplifies a method for classifying obstruction types in a cameraobstruction detecting system, according with an embodiment of thepresent invention; and

FIG. 8 is a schematic flow diagram illustration of selecting day andnight modes for a camera obstruction detecting system, in accordancewith an embodiment of the present invention.

FIG. 9 a shows an example image of rain drops on the windshield asviewed by the secondary camera of a camera obstruction detecting system,in accordance with an embodiment of the present invention.

FIG. 9 b shows the example of the rain drops on the windshieldillustrated in FIG. 9 a, as viewed by the primary camera of a cameraobstruction detecting system, in accordance with an embodiment of thepresent invention.

FIGS. 10 a, 10 b and 10 c illustrate a raindrop and its tri-points thatare a distinctive feature of raindrops.

FIG. 11 shows, by way of example, a series of graphs characterizing thetri-points that are a distinctive feature of raindrops.

FIG. 12 a shows an example image of an opaque obstruction on thewindshield, as viewed by the secondary camera of a camera obstructiondetecting system, in accordance with an embodiment of the presentinvention.

FIG. 12 b shows the example of the opaque obstruction on the windshieldillustrated in FIG. 12 a, as viewed by the primary camera of a cameraobstruction detecting system, in accordance with an embodiment of thepresent invention.

FIG. 13 shows, by way of example, the drop in image contrast of primaryimages, caused by a smeared/squashed insect.

FIG. 14 shows, by way of example, the differences in typical edgeprofiles of a rain drop edge segment, light mud edge segment and thickmud edge segment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is a system mounted on a vehicle for detecting anobstruction on an external surface of a windshield of the vehicle. Aprimary camera typically used for a driver's warning system and/or for avehicle control system, is mounted inside the vehicle behind thewindshield. The primary camera is configured to acquire images of theenvironment through the windshield. A secondary camera is focused on asurface of the windshield. Due to the angle of the windshield a portionof the secondary camera is focused on the external surface of thewindshield, and a portion of the secondary camera is focused on theinterior of the window. The secondary camera operates to image theobstruction. A portion of the windshield, i.e. windshield region issubtended respectively by the field of view of the primary camera andthe field of view of the secondary camera. The primary and secondarycameras work in conjunction with each other in order to assure thatthere are no obstructions blocking the primary camera, and to minimizeunnecessary image processing of the secondary camera (e.g., in the casewhereby the primary camera recognizes bright points of light fromon-coming vehicles and notifies the secondary camera that the brightspot is indeed a light source and not moisture or some other obstructionon the windshield.)

Before explaining embodiments of the invention in detail, it is to beunderstood that the invention is not limited in its application to thedetails of construction and the arrangement of the components set forthin the host description or illustrated in the drawings.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art of the invention belongs. The methods and examples providedherein are illustrative only and not intended to be limiting.

By way of introduction, the principal intentions of the presentinvention include detecting and classifying different obstructingsubstances on the windshield.

An aspect of the present invention is to distinguish between the variouscauses of image impairment since different causes of image impairmentrequire a different response from the driver. In particular (a) raindrops and snow flakes prompt activation of the wipers, (b) dirt promptsthe activation of the wipers with fluid and if the dirt persists, thedriver is optionally prompted to manually clean the windshield, (c)smoke and dust particles collected on the inside of the windshield mightrequire special service, (d) fog prompts lighting of fog light and (e)condensation prompts activating a defogger. Thus it is important to beable to distinguish between the different causes of image impairment.When an obstruction is detected, a low visibility signal may begenerated indicating possible system unavailability due to imageimpairment.

It is another aspect of the present invention to integrate the detectionand identification of the obstructions on the windshield with thefunction of the primary camera, i.e. a driver's warning system and/orfor a vehicle control. It should be noted that all obstructions on thewindshield very similar in the images acquired by the primary camera,the primary camera being typically focused on objects outside thevehicle. Obstructions as viewed by the primary camera appear as blurredimages, because the windshield and the obstructions on an external andinternal surface of the windshield are significantly out of focus. Thusthe primary camera is preferably not used to determine the cause of thevisibility impairment. FIG. 9 a shows an example image of rain drops(502 and 504) on windshield 12 as viewed by the secondary camera. Therain drops are imaged by the primary camera as a blurred patch 512 and514, shown in FIG. 9 b, while imaging a distant checkerboard-like targetFIG. 12 a shows an example image of dirt (530) on windshield 11 asviewed by the secondary camera. The dirt is imaged by the primary cameraas a blurred patch 532, shown in FIG. 12 b, while imaging a distantcheckerboard-like target.

FIG. 2 illustrates a vehicle having a camera obstruction detection andclassification system, in accordance with an embodiment of the presentinvention. FIG. 2 depicts a host vehicle 10 that travels on a roadsurface 20, having a camera obstruction detection system 30, including:a primary camera 32; a secondary camera 33; and a processing unit 34.Primary camera 32 is mounted on host vehicle 10 viewing the environmentthrough windshield 12 with a field of view typically of 30-50 degrees.Secondary camera 33 is mounted on host vehicle 10 and focuses onwindshield 12 preferably over a windshield region subtended at least inpart by the field of view of primary camera 32.

FIG. 3 is a perspective view of an example of a camera obstructiondetecting system 30, having a primary camera 32 and a secondary camera33 situated side by side. Windshield region 43 of secondary camera 33 isat least overlapping windshield region 42 of primary camera 32.

FIG. 4 is a perspective view that exemplifies a camera obstructiondetection system 30 where primary camera 32 and secondary camera 33 areset one below the other. Windshield region 43 of secondary camera 33 isat least overlapping windshield region 42 of primary camera 32.

The present invention is not limited to a side-by-side embodiment or atop-down embodiment. The system may be mounted on a front or rearwindshield, as well as on a side window—or behind any glass surface, forexample behind the glass enclosure of the headlamps or tail lamps of thevehicle.

FIG. 5 is a close up view illustrating a camera obstruction detectionsystem 30 where primary camera 32 and secondary camera 33 are situatedone below the other, behind windshield 12. Windshield 12 has lightobstructing substances 50 stuck onto windshield 12 external surfaces andblocks some of the light returning to primary camera 32. Secondarycamera 33 is focused on an external surface of windshield 12 preferablyin region subtended by the field of view of primary camera 32.Typically, system 30 is continuously monitoring windshield 12, analyzingsecondary images. In some embodiments, the analysis of identifying anobstruction 50 is triggered by the primary vision processor 60.

Often, the optical axis secondary camera 33 is not completelyperpendicular to windshield 12 surface. Due to the thickness ofwindshield 12 and the finite depth of field of secondary camera 33, insome parts of the image, typically the lower parts of the image, theinner surface of the windshield 12 is most in focus. Typically, in theupper parts of the image, the outer surface of windshield 12 is most infocus. Having the ability to also focus secondary camera 33, in parts ofthe secondary image, on the inner surface of windshield 12 allows system30 to determine, in a case of fog, condensation or smoke residue (whichthe camera picks up as texture) whether it is outside the vehicle 10 oron the inner surface of windshield 12. Having the ability to also focussecondary camera 33, in parts of the secondary image, on the innersurface of windshield 12 allows system 30 to also detect otherobstructions on the inner surface of windshield 12, such as accumulatedsmoke particles, fog or dust, and to determine where the obstruction ison the inside or outside of windshield 12. One such situation where itis necessary to check the inside of windshield 12 is when there is fogon windshield 12. The driver might have turned on the defogger to clearwindshield 12, and yet the region in front of the camera might still notbe cleared of the fog or frost. By checking if the fog is on the insideor outside of windshield 12, the system will be able to determine theproper corrective action, e.g. whether to signal for the heater,defogger or low visibility mode.

FIG. 6 is a schematic flow diagram showing data and decision flows insystem 30, in accordance with an embodiment of the present invention. Ina vision system 30 mounted on host vehicle 10, primary camera 32 viewsthe scene in front of primary camera 32 through windshield 12 as inprior art vision system 80. Often, light passing through windshield 12is at least partially blocked, attenuated, scattered or absorbed byobstructions 50 on the external surface of windshield 12, beforereaching primary camera 32. Primary camera 32 acquires images andtransmits (step 132) the acquired images to primary vision processor 60to be processed. Secondary camera 33 transmits (step 133) the acquiredimages to a high frequency presence decision module 134 for processing.

Processor 34 detects substances 50 adhering to an external surface ofwindshield 12, and analyzes the image deterioration if any caused bydetected light obstructing substances 50, on the images acquired byprimary camera 32. The image deterioration caused by light obstructingsubstances 50 is considered negligible if the obstructed light has nomeasurable influence on the image quality from images from primarycamera 32. The image deterioration when measured optionally causessystem 30 to activate a low visibility mode, if the obstructed lightreduces the performance of primary camera 32, but still executes part ofthe tasks. The image deterioration caused by light obstructingsubstances 50 can even cause primary camera 32 to stop function.

In vision systems, where the image analysis is at least partially basedon edge detection, such edges are represented in the images by highspatial frequencies. If windshield 12 is sufficiently clean, edges withsignificant gradient do not show in the images acquired by secondarycamera 33 if obstructions 50 are situated on a windshield region ofwindshield 12, then processing unit 34 determines the area on windshield12 containing obstructions 50. The corresponding area in images acquiredby primary camera 32 is also analyzed for high spatial frequencies. Ifthe gradient of the edges detected is lower than some threshold, system30 activates low visibility mode, having determined that obstructions 50are present on a windshield region of windshield 12.

Referring back to FIG. 6, secondary camera 33 transmits (step 133) theacquired images to a high frequency presence decision module 134 to beprocessed. If processing unit 34 detects in images acquired by secondarycamera 33, in step 134, high spatial frequencies, representing edges,and in respective regions of respective images obtained by primarycamera 32 no high spatial frequencies are observed in step 135, decisionstep 136 activates a low visibility mode, in step 137. Obstructions 50detected by processor 34 in the images acquired by secondary camera 33,are classified in step 300 and actions are taken in step 301 accordingto the type of obstructions 50 detected and classified.

FIG. 8 shows a schematic flow diagram outlining an algorithm 300 thatexemplifies a method for classifying obstruction 50 types in a cameraobstruction detecting system 30, according with an embodiment of thepresent invention. Obstructions 50 are detected by secondary camera 33,being focused on windshield 12.

As a preliminary step, system 30 maps out false edges in the secondaryimage obtained in step 305, for example, a distant light source appearsas a focused point of light in the primary image. It is well known fromoptics, that in a camera that is focused on the near distance (thesecondary camera in the present invention), this point of light willappear as a disc with sharp edges. Such discs with sharp edges must beeliminated (step 306) from further analysis. The masked out secondaryimages are analyzed in step 310 for high frequencies representing sharpedges. If no edges were detected, Images obtained in step 305 areanalyzed in step 320 for blurry regions. If no blurry regions weredetected in the primary image, algorithm 300 assumes windshield region43 is clear. If low visibility mode was active, low visibility mode isturned off in step 325 and if the wipers were active the wipers areturned off in step 335. If blurry regions were detected in the primaryimage in step 320, fog, smoke (which is granular) or condensationsituation or the like are assumed to persist; low visibility mode isactivated in step 330 but if the wipers were activated, the wipers areturned off in step 335.

If in step 310 at least one edge is detected, system 30 proceeds into aclassification procedure to classify 311 the detected obstruction 50.Images 110 obtained in step 305 are further analyzed in step 312 todetermined id the obstruction is a crack in windshield 12. If theobstruction is a crack, the crack edges are masked out from thesecondary image, and system 30 proceeds to obtain the next image in step305. If the classification fails in step 312, system 30 looks forobjects with bright, dark, and grey spots in step 340. Objects withbright, dark and grey spots characterize rain drops. Rain may bestationary or seen as moving depending on whether the vehicle isstationary or moving. If one or more moving objects with both bright anddark spots are detected in step 340, rain is assumed and the wipers areactivated in step 380.

If in step 340 no raindrops are detected, further analysis action istaken in step 350 for detecting opaque objects such as thick mud. If instep 350 no opaque objects were detected, further analysis action istaken in step 355 for detecting semi-opaque objects such as frost ordust or smoke residue. If in step 350 no semi-opaque objects weredetected, algorithm 300 assumes obstruction 50 represents an unknownobstruction 50. The unknown obstruction 50 might be a temporaryobstruction, or a failing sensor, etc. Primary vision system 60 may benotified in step 357 and system 30 proceeds to obtain the next image instep 305.

If in step 350 one or more opaque object was detected an attempt is madeto remove the detected opaque object by activating in step 390 the mistfor the wipers and in step 380 the wipers themselves. If in step 360after a preset number of attempts to remove obstructions 50 is surpassedand obstructions 50 sustains, primary system processor 60 (FIG. 6) isdeactivated and the driver is preferably notified.

If in step 355 frost is detected an attempt is made to remove the frostby activating in step 356 the defroster. If smoke residue is detectedthe driver may be notified to remove the smoke residue.

The following gives the characteristics of various types of visualobstructions 50 and methods to detect them and remove them:

-   -   (a) As a preliminary step, system 30 maps out false edges in the        secondary image, for example, a distant light source appears as        a focused point of light in the primary image. It is well known        from optics, that in a camera that is focused on the near        distance (the secondary camera in the present invention), this        point of light will appear as a disc with sharp edges. Such        discs with sharp edges must be eliminated from further analysis.        If primary camera 32 identifies a point of light, secondary        camera 33 is notified exactly where this point of light is        located. Secondary camera 33 then recognizes the corresponding        disc of light with a known radius to have been produced by a        light source and ignores the disc edges. System 30 looks for        rings of a focus typical of distant point of light. The radius        can be determined empirically or calculated by standard methods        of optics, see for example MIT press, Berthold Horn, Robot        Vision, included herein by reference for all purposes as if        entirely set forth herein.        -   the radius can be determined empirically: the vertical and            horizontal derivatives of the image are computed and are            combined to a derivative magnitude image:

$\begin{matrix}{I_{mag} = {\sqrt{I_{x}^{2}} + I_{y}^{2}}} & (1)\end{matrix}$

-   -   -   a binary map of all pixels whose edge is above a certain            threshold is computed. The threshold can be fixed or data            dependent such as N*std over the mean, etc.        -   In an embodiment of the present invention, bright spots are            detected in the primary image for identifying a distortion            50 at night time. For each detected bright spot, a ring of            radius R and thickness T is dropped from further            consideration (see equation 1). Steps are taken again to            compute a derivative magnitude image. Then, the number of            pixels C in I_(mag) that are above a certain threshold are            counted. The threshold can be fixed or data dependent such            as N*std over the mean, etc. If C is greater than some            threshold, then an obstruction is detected.        -   As an additional preliminary step, system 30 eliminates the            areas which have remained on window 12 for a period of time.            These could indicate ‘poc-marks’ on the window caused by            small stones or some other small object which left a            ‘bullet-like’ hole without actually having cracked window            12.

    -   (b) Test for frost, dust and/or smoke particles: system 30 uses        a RBF SVM (Support Vector Machine with Radial Basis Function        Kernel), see for example: Cambridge University Press, Cambridge,        UK, Cristianini and Shawe-taylor, An Introduction to Support        Vector Machines and other kernel based learning methods included        herein by reference for all purposes as if entirely set forth        herein; or in Scholkopf, Burges and Smola Eds., Advances in        Kernel Methods: Support Vector Learning, The MIT Press,        Cambridge, Mass., included herein by reference for all purposes        as if entirely set forth herein.        -   Frost and smoke-residue/dust (hereinafter referred to as            smoke) each have their specific textures easily            distinguishable to the human eye, and typically cover large            areas A classifier trained on a predetermined window patch            size, classifies an obstruction 50 as being a frost or smoke            residue. System 30 classifies all patches (for example of            size 16×16) in the image which do not include any of the            previously detected obstacles using a template based            classifier (such as RBF SVM) taught with examples of            frost/smoke residue/clear/other, where ‘other’ includes            examples of mud/rain/light mud etc. It should be noted that            there are other techniques for classifying textures which            are well published and can be sued to replace the            classification step. Since the classifiers are binary            classifiers, system 30 can use, for example, four separate            sub-classifiers to classify a patch in the secondary image.            -   a. frost against smoke residue/clear/other;            -   b. smoke against frost/clear/other;            -   c. clear against frost/smoke residue/other; and            -   d. other against frost/smoke residue/clear.        -   The patch, in this example, is given the classification of            the sub-classifier with the highest positive score. If all            scores are negative it is classified as ‘unknown’.        -   System 30 then computes the total area of frost by taking            the union of all patches classified as frost, and the total            area of smoke by taking the union of all patches classified            as smoke residue. If total frost area exceeds a certain            threshold or the number of patches classified as frost            exceeds a certain threshold, then the condition is            classified as ‘possible frost’. If total smoke area exceeds            a certain threshold or the number of patches classified as            smoke exceeds a certain threshold, then the condition is            classified as ‘possible smoke residue’.        -   If both ‘possible frost’ and ‘possible smoke residue’            conditions exist, system 30 acts as if just frost exists,            including: notifying the driver, activating a defrost            mechanism, notifying primary vision system 60. If only the            ‘possible smoke residue’ condition exists, system 60 must            determine if the residue is really smoke residue on            windshield 12 internal surface or some other fine residue on            the external surface of windshield 12 (the image texture is            very similar to that of fine pollen). To detect if the smoke            residue texture is on the internal surface or external            surface of windshield 12, system 30 compares the spatial            frequencies of the texture in the upper and lower parts of            the secondary image. Since windshield 12 is at an angle to            the camera neither inner or out surfaces are completely in            focus throughout the image. The optics can be designed such            that the lower in the image, external surface is in focus            and the internal surface is more in focus. In particular,            the optics can be designed such that the transition line            cuts through typical areas of smoke residue build up. If            smoke buildup exists on the inner surface it might require            service.        -   If ‘possible smoke residue’ conditions exist on the internal            surface of windshield 12, appropriate actions are taken in            step 301, for example: inform the driver: ‘service is            required’.

    -   (c) Test for cracks: system 30 computes a derivative magnitude        image, as in equation 1. If connected edges form a first long        edge segment that are longer than a certain threshold N and the        curvature of the edge segment is less than a certain threshold T        and a second edge segment which is generally parallel to the        first segment than system 30 classifies the obstruction 50 as        being a crack in windshield 12. Furthermore, the crack segment        points can be added to binary mask, not to be used for rain and        other detections. If a second edge segment which is generally        parallel to the first segment is not found, the obstruction 50        may be a sticker or flyer attached to windshield 12. If crack in        windshield 12 or any other long obstruction 50 is identified,        appropriate actions are taken in step 301, for example: notify        the driver and/or low visibility mode is activated.

    -   (d) Test for rain drops: In daytime, system 30 looks for strong        horizontal edge components with dark patch above bright patch        surrounded by a gray region FIG. 9 a shows an example image of        rain drops (502 and 504) on windshield 12 as viewed by secondary        camera 33 Rain drops 502 and 504 are imaged by primary camera 32        as corresponding blurred patches 512 and 514, shown in FIG. 9 b,        while imaging a distant checkerboard-like target. System 30        computes a bounding box which includes both light and dark        patches. System 30 then computes histograms of image gradient        orientations for the four quadrants of the bounding box, and        uses pattern, for example, a RBF SVM classifier trained on image        gradient orientations, to classify as a rain drop.        -   In an embodiment of the present invention, system 30 looks a            distinctive feature of raindrops: a junction of the three            regions with different shades: dark, bright and grey. We            hereinafter refer to these special junction points as            tri-points, which typically positioned at either the on the            left or the right of the rain drop.        -   FIGS. 10 a, 10 b and 10 c illustrate a typical raindrop 500            with tri-points 505 and 507 that are a distinctive feature            of raindrops. The surrounding windshield 12 is a bit darker            than the bright region 511 and lighter than the dark region            509 of a rain drop 500. The junction of the three regions            with different shades: dark 509, bright 511 and grey 510 are            highlighted by circle 503 in FIG. 10 b. The junction is            referred to as a tri-point. To locate such tri-points in the            image system 30 analyzes all edge points in the image. For            each edge point system 30 looks at the image brightness of            the points along a circle of radius R (e.g. R=5 pixels)            around the edge point being tested, such as circle 503. FIG.            10 c shows the left tri-point 505 and right left tri-point            507. FIG. 11 shows examples of the brightness profiles on            circles surrounding edge points that are tri-points, edge            points that are only between two regions such as the edge of            a patch of frost, and around points that are not on an edge.            Tri-points can be subclassified into left and right            tri-points. To classify, system 30 finds the best match to            templates such as the templates shown in the FIG. 11, using            nearest neighbor with a normalized correlation distance            measure. The left and right tri-points (FIG. 11) that lie on            the same edge segment (edge points, FIG. 11) are matched.            For each matching pair left and right tri-points, that            define a raindrop, system 30 may also verify that the edge            segment has consistent sign along the edge elements and that            the generally enclosed edge segment contains a dark region            above a bright region. Also, the distance between points is            below a certain value which depends on the specific optics.            Another criterion that can be used is that the angle of the            straight line joining the two tri-points is less than 45°            from horizontal. One can estimate the area of a rain drop by            taking the distance between the two tri-points, squared. If            the number of rain drops detected exceeds a threshold or the            total area of rain drops exceeds a threshold, then the            windshield wipers are activate (step 380).        -   For the sake of clarity, it is noted that these methods may            not detect all rain drops in every frame but these methods            have a detection rate of above 80%, which is sufficient for            this application.        -   At night time the scenery is different. Bright spots are            detected in the primary image. For each detected bright            spot, a ring of radius R and thickness T is matched in the            secondary images. System 30 can then look for circles in the            secondary image and see if there are breaks in the edge,            indicating some interference on the external surface of            windshield 12. If rain drops are detected, appropriate            actions are taken in step 301, for example: activating the            wipers in step 380.

    -   (e) Test for snow flakes: system 30 uses, for example, RBF SVM        classifier trained on a predetermined size patch, to classify an        obstruction 50 as being a snowflake. If snow flakes are        detected, appropriate actions are taken in step 301, for        example: activating the wipers in step 380.

    -   (f) Test for thick mud or light mud or squashed insect: light        mud, squashed insect, bird droppings and the like, appear quite        similar in the image. FIG. 12 a shows an opaque obstruction 530        on windshield 12 as viewed by secondary camera 33. FIG. 12 b        shows the same example as viewed by primary camera 32.        Obstruction 530 is imaged by primary camera 32 as blurred patch        532. FIG. 13 shows, by way of example, the drop in image        intensity of primary images, caused by a smeared/squashed        insect. The checker-board like image shows clearly a drop in        intensity in the obstructed region. The main difference is the        size of the affected area which is usually limited for insects        and extensive for light mud. Since system 30 response is the        same (activate wipers with fluid+notify primary vision system of        affected regions) thick mud or light mud or squashed insect are        detected in the same module and treated the same.        -   System 30 locates dominant edges (i.e., edges appearing in            both high and low levels of the image resolution). The edges            may be tracked over time (a few frames) to determine if the            edges are stationary, and classifies the edge profile into            thick mud, light mud and rain drop profiles by, for example,            template matching (nearest neighbor, normalized            correlation). FIG. 14 shows, by way of example, the            differences in typical edge profiles of a rain drop edge            segment, light mud edge segment and thick mud edge segment.            The light mud edge segment is of the lowest contrast. If            edges matching the light mud or thick mud profiles persist,            system 30 assumes it represents an opaque obstruction 50            such as dirt, mud, snow, smeared insect, bird droppings and            the like.        -   If no match is found, system 30 can further analyze            obstruction 50, which is a blob. System 30 grows the blob by            up to 3 pixels, to nearest edge point. Blobs that include            edge points that more than 50% of the edge points are            masked—are discarded. System 30 then classifies edge points            on the remaining blobs. If 50% or more of the edge points            are classified as thick mud, then the blob is classified as            thick mud. If 50% or more of the edge points are classified            as light mud, then the blob is classified as light mud. If            total area of the blob is above a certain threshold, system            30 attempts to remove obstruction 50 by activating the            wipers in step 380 along with mist, which is activated in            step 390. If the condition persists the driver is notified.            System 30 notifies primary vision system 60 of the extent of            the obstruction.        -   For all overlapping sub-patches in the image (for example, a            sub-patch can be of size 10×10 pixels), system 30 counts the            number of edge pixels that are classified as light mud, and            the number of edge pixels classified otherwise. If the            number of edge pixels that are classified as light mud is            greater than the rest of the edge pixels by some threshold,            then system 30 classifies the blob as light mud. The edges            are tracked over time (a few frames) to determine if the            edges are stationary. If edges matching the light mud or            thick mud profiles persist, system 30 assumes it represents            an opaque obstruction 50 such as dirt, mud, snow, smeared            insect, bird droppings and the like. Attempt to remove them            by activating the wipers in step 380 along with mist,            activated in step 390.

    -   (g) For each region with high frequencies not classified in the        above mentioned tests, system 30 locates dominant edges (i.e.,        edges appearing in both high and low levels of the image        resolution), tracks the edges over time and determine if the        edges are stationary. If a region with high frequencies        persists, system 30 assumes it represents a solid obstruction        and computes the size of the obstruction 50. If the computed        area is larger than some threshold, an attempt to remove is done        by activating the wipers in step 380 along with mist, which is        activated in step 390.

FIG. 7 is a schematic flow diagram illustration of selecting day andnight modes for a camera obstruction detecting system 30, in accordancewith an embodiment of the present invention. In an initial step 150,system 30 determines if ambient lighting is that of day time or nighttime and sets system 30 parameters accordingly, in either step 151 orstep 152. In the next step 300, system 30 locates and analyzes theimages obtained by secondary camera 33 for regions of high spatialfrequency, classifies detected obstructions 50 and selects for each typeof obstructions actions to be taken. The analysis for high spatialfrequencies can be done by using Fourier transform, by comparing thegradient strength in images at various levels of image resolution and/orby any other method known in the art. If high frequencies are detectedin step 310, corresponding regions in corresponding images obtained byprimary camera 32 are also analyzed for high spatial frequencies. If atleast one of the corresponding regions in corresponding images obtainedby primary camera 32 lack high frequencies, low visibility analysis 300proceeds.

Secondary camera 33 is not performing time critical processing and thus,secondary camera 33 can be optionally used for other tasks, tosupplement primary camera 32. For example, secondary camera 33 can beused as an ambient light sensor or as a sensor for the gain control ofprimary camera 32.

In another embodiment of the present invention, system 30 also includesa light source, for example a LED, mounted, for example, insidesecondary camera 33 mount. At dark nights, the light source flashesinstantaneously, at a low frequency rate, such that the pulsing does notinterfere with other operations of primary camera 32. When flashing, thelight source illuminates windshield 12 and secondary camera 33, which issynchronized with the timing of the flashing, is imaging the litwindshield 12.

The invention being thus described in terms of embodiments and examples,it will be obvious that the same may be varied in many ways. Suchvariations are not to be regarded as a departure from the spirit andscope of the invention, and all such modifications as would be obviousto one skilled in the art are intended to be included within the scopeof the following claims.

What is claimed is:
 1. A system mounted on a vehicle for classifying atleast one obstruction on a surface of a window of the vehicle, thesystem comprising: a secondary camera adapted for mounting inside thevehicle, wherein said secondary camera is adapted to focus on at least aportion of the surface of the window and acquire a secondary image ofthe at least one obstruction when present on the surface; and aprocessor adapted to process said secondary image and for classificationof the at least one obstruction; and a primary camera adapted formounting inside the vehicle, wherein the primary camera is configured toacquire a plurality of primary images of the environment through thewindow, wherein a region of the window is subtended by respective fieldsof view of said secondary camera and said primary camera, and whereinsaid processor is configured to process at least one of said primaryimages and to trigger said classification of the at least oneobstruction responsive to deterioration of image quality in at least oneof said primary images.
 2. The system of claim 1, wherein information isextracted from said at least one primary image to selectably eitherenhance said classification of said at least one obstruction oreliminating false identification of an obstruction.
 3. The system ofclaim 1, wherein said deterioration of image quality is caused by saidat least one obstruction.
 4. The system of claim 1, further comprising:an activation mechanism operatively attached to said processor, whereinsaid activation mechanism causes cleaning fluid to be sprayed on thewindow and windshield wipers to be activated when said processoridentifies the at least one obstruction as including dirt on an externalsurface of the window while the external surface is dry.
 5. The systemof claim 1, further comprising: an activation mechanism operativelyattached to said processor, said activation mechanism activatingwindshield wipers when said processor identifies the at least oneobstruction on an external surface of the window.
 6. The system of claim1, wherein said processor is adapted to initiate a task to remove saidat least one obstruction.
 7. The system of claim 6, wherein said task isselected from the group of tasks consisting of: notification of thedriver of the vehicle, disabling the system, activating wipers with orwithout cleaning fluid activating defogging, and turning on fog lights.8. The system of claim 1, wherein said secondary camera is adapted toalso focus on at least a portion of an internal surface of the windowand wherein said processor identifies the at least one obstruction whenpresent on said at least a portion of the internal surface.
 9. Thesystem of claim 1, wherein said secondary camera is further adapted toperform a task selected from the group of applications consisting of:ambient light sensing, and gain control sensing.
 10. The system of claim1, further comprising a pulsing light source, wherein said pulsing lightsource is capable of illuminating said region wherein secondary imageacquisition by said secondary camera is synchronized with activatingsaid pulsing light source.
 11. A method performable by a systemmountable inside a vehicle, the system including a primary camera and asecondary camera both connectible to a processor, the method comprisingthe steps of: enabling focusing the secondary camera on a surface of thewindow, to provide at least one secondary image of said surface;secondary processing said at least one secondary image therebyidentifying when present on said surface at least one obstruction;enabling acquiring a plurality of primary images of the environmentthrough the window, wherein a region of the window is subtended byrespective fields of view of the secondary camera and the primarycamera; and primary processing said at least one primary image therebydetecting at least one obstruction, when present on said surface, saidat least one obstruction causing a deterioration of image quality insaid at least one primary image, wherein said deterioration of imagequality in said at least one primary image triggers said secondaryprocessing of said at least one secondary image and said classifyingthereby the at least one obstruction.
 12. The method of claim 11,wherein said secondary processing includes edge detection of said atleast one obstruction.
 13. The method of claim 11, wherein saidsecondary processing of said at least one secondary image usesinformation extracted from said at least one primary image to selectablyeither enhance said identifying said at least one obstruction oreliminating false identification of an obstruction.
 14. The method,according to claim 11, further comprising a step of: upon determining alevel of ambient light, setting a mode of the system to selectablyeither day mode, dusk mode or night mode.
 15. The method, according toclaim 14, when determining said level of ambient light as being of duskmode or night mode, turning on the headlights.
 16. The method of claim11, further comprising the step of: initiating at least one task toremove said at least one obstruction.
 17. The method of claim 16,wherein said at least one task is selected from the group of tasksconsisting of: notification of the driver of the vehicle, disabling thesystem, activating windshield wipers, spraying fluid on windshield andactivating defogging of a window.
 18. The method of claim 11, whereinsaid primary processing of said at least one primary image includes atleast one other vehicular vision processing step selected from the groupconsisting of: detecting lane markings in a road, detecting pedestrians,detecting vehicles, detecting obstacles, detecting road signs, lanekeeping, headway keeping and headlights control; and wherein said atleast one other vehicular vision processing step is turned off.
 19. Themethod of claim 11, wherein said identifying of said at least oneobstruction includes classifying said at least one obstruction, using aSupport Vector Machine (SVM) technique.
 20. The method of claim 19,wherein said identifying of said at least one obstruction includesclassifying use of support vector machine (SVM) with Radial BasisFunction kernel techniques.
 21. The method of claim 11, furthercomprising the step of: illuminating said window with a pulsing lightsource, wherein said secondary camera is operated synchronously withsaid pulsing light source.